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EXECUTIVE  SUMMARY 


Automated  controllers  are  becoming  an  integral  part  of  many  complex 
systems.  One  such  controller  is  an  automated  detector  and  tracker  (ADT)  which 
aids  an  operator  in  detecting  and  tracking  the  location  of  targets  such  as  ships, 
tanks,  or  aircraft.  To  study  human  use  of  an  ADT,  an  Automated  Detection  and 
Tracking  Simulation  (ADTS)  system  has  been  implemented.  The  ADTS  is  a 
modification  of  an  Automated  Tracking  Simulation  (ATS)  that  has  been  used  to 
study  the  use  of  an  automated  tracker  (AT)  as  a  function  of  its  reliability  and  task 
difficulty.  With  both  systems,  the  user's  task  is  to  detect  and  track  the  position  of 
targets.  With  the  ATS,  the  user  has  the  option  of  assigning  some  or  all  targets  to 
an  automated  tracker  (AT)  which  mimics  the  user  by  trying  to  update  the 
position  of  targets  that  it  is  responsible  for.  This  capability  has  the  overhead  of 
having  to  assign  and  deassign  targets  every  time  the  AT  fails  to  update  a  target, 
but  it  gives  the  user  ultimate  control  over  the  task.  The  ADTS,  in  addition  to 
tracking  existing  targets,  has  the  ability  to  add  targets  to  the  display. 

The  purpose  of  the  current  experiment  was  to  determine  if  task  differences 
between  the  two  systems  affected  performance.  In  the  ADTS,  the  user  does  not 
have  the  option  of  handling  some  targets  manually.  This  difference  makes  the 
user  more  of  a  system  monitor  than  an  active  participant,  but  it  also  reduces  the 
number  of  actions  required  to  handle  ADT  errors.  Task  differences  were  assessed 
by  comparing  performance  on  the  two  systems  when  their  capabilities  were 
identical.  Performance  on  the  ATS  was  compared  with  performance  on  the 
ADTS  with  the  detection  threshold  set  so  that  either  no  targets  were  added  or  no 
non-targets  were  added.  The  results  indicated  no  differences  in  hit  rate  or 
response  time  across  the  three  conditions.  However,  miss  rate  decreased 
significantly  and  false  alarm  rate  increased  significantly  between  the  ATS 
condition  and  the  second  ADTS  condition.  The  increase  in  false  alarms  was  due 
primarily  to  an  increase  in  manual  false  alarms  and  was  attributed  to  a  change  in 
strategy.  However,  it  was  not  clear  if  the  change  in  strategy  was  due  to  the  change 
in  task  or  to  the  experience  level  of  the  participants.  All  of  the  participants  in  the 
ADTS  task  had  previously  completed  several  sessions  with  the  ATS. 
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ABSTRACT 


The  experiment  in  this  report  compared  performance  on  a  target  tracking 
task  using  an  Automated  Tracking  Simulation  (ATS)  system  with  performance 
on  the  same  task  using  an  Automated  Detection  and  Tracking  Simulation 
(ADTS)  system.  With  both  systems,  the  user's  task  is  to  detect  and  track  the 
position  of  targets  by  comparing  the  location  and  strength  of  new  signals  with 
the  location  of  existing  targets.  With  the  ATS  the  user  has  the  option  of  assigning 
some  or  all  of  the  targets  to  an  automated  tracker  (AT).  The  ADTS,  in  addition  to 
tracking  existing  targets,  has  the  ability  to  add  targets  to  the  display.  The  purpose 
of  the  current  experiment  was  to  determine  if  task  differences  between  the  two 
simulations  affected  performance.  The  results  indicated  no  differences  in  hit  rate 
or  response  time  across  the  three  conditions.  However,  miss  rate  decreased 
significantly  and  false  alarm  rate  increased  significantly  between  the  ATS 
condition  and  the  ADTS  condition  with  the  detection  threshold  set  to  the 
maximum  strength  for  non-target  signals.  This  difference  was  attributed  to  a 
change  in  strategy.  However,  it  was  not  clear  if  the  change  in  strategy  was  due  to 
the  change  in  task  or  the  experience  level  of  the  participants.  All  of  the 
participants  in  the  ADTS  task  had  previously  completed  several  sessions  with 
the  ATS. 
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INTRODUCTION 


Background 

Automated  controllers  and  decision  aids  are  becoming  an  integral  part  of 
many  complex  computer-based  systems.  They  are  implemented  with  the  aim  of 
reducing  human  error  and  workload  and  improving  system  performance.  One 
such  controller  is  an  Automated  Tracker  (AT).  It  tracks  the  position  of  targets 
(other  ships,  submarines,  etc.)  by  comparing  the  location  and  strength  of  new 
signals  with  the  location  of  existing  targets.  If  the  position  of  a  signal  is  within  a 
certain  area  surrounding  an  existing  target,  the  AT  associates  that  signal  with  the 
target,  updating  the  position  of  the  target. 

If  the  objectives  for  automated  controllers  are  to  be  realized,  it  is  important 
to  understand  how  and  when  people  use  them  and  the  impact  of  those  decisions 
on  overall  system  performance.  To  this  end,  a  simulation  of  a  generic  target 
tracking  task  in  a  naval  environment  was  developed.  The  user's  task  is  to  detect 
and  then  update  the  position  of  a  series  of  targets  over  time.  He  or  she  can  carry 
out  the  task  manually  or  assign  some  or  all  of  the  targets  to  be  tracked  by  an 
automated  tracker.  Previous  experiments  (McFadden,  Giesbrecht,  &  Gula,  1997) 
have  examined  the  use  of  the  automated  tracker,  percentage  of  targets  tracked, 
and  types  of  errors  as  a  function  of  the  actual  and  perceived  reliability  of  the 
tracker. 

Many  automated  tracking  systems  include  an  automated  detection 
function  as  well  allowing  the  automated  system  to  both  detect  and  track  targets. 
Adding  an  automated  detector  changes  the  nature  of  the  task  somewhat.  Users 
can  no  longer  choose  to  give  control  of  a  target  to  the  automated  tracker  or  to 
handle  it  manually.  Their  role  shifts  even  more  towards  a  monitor  of  the 
automated  system  rather  than  an  active  participant  in  the  task.  To  study  the 
impact  of  the  inclusion  of  an  automated  detection  capability,  an  automated 
detection  fimction  was  added  to  the  ATS.  This  modified  version  of  the  ATS  is 
called  the  Automated  Detection  and  Tracking  Simulation  (ADTS). 

As  with  a  real  system  employing  an  automated  detection  capability,  the 
user's  task  on  the  ADTS  differs  from  that  of  the  ATS.  With  the  ATS,  the  user  had 
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the  option  of  assigning  a  target  to  the  AT  or  tracking  it  manually.  If  the  AT  failed 
to  update  the  position  of  a  target,  the  user  had  to  deassign  the  target  (take  it  away 
from  the  AT),  update  the  position  of  the  target,  and  if  desired,  reassign  it  to  the 
ATk  With  the  ADTS,  there  is  no  longer  any  requirement  to  assign  and  deassign 
targets.  Both  the  human  and  the  automated  system  can  potentially  add  and 
associate  any  of  the  available  signals  every  time  new  information  becomes 
available.  Thus,. it  is  not  possible  to  prevent  the  AT  from  trying  to  handle  certain 
targets.  However,  the  user  no  longer  has  the  time  cost  of  deassigning  targets  that 
the  AT  fails  to  update.  In  order  to  separate  out  the  effect  of  adding  an  automated 
detection  capability  from  the  change  in  the  task,  it  is  necessary  to  replicate  some 
of  the  conditions  run  on  the  ATS  on  the  ADTS. 


Current  study 

This  experiment  compared  performance  on  a  target  tracking  task  using  the  ATS 
with  performance  on  the  same  task  using  the  ADTS.  Identical  scenarios  were 
used  on  the  two  tasks  and  the  automated  tracker  parameters  were  set  so  that  the 
AT  would  be  highly  reliable  (McFadden,  Giesbrecht  &  Gula,  1997).  With  the 
ADTS,  the  detection  threshold  was  set  so  that  automated  detector  would  not 
detect  any  targets  or  so  that  it  would  never  detect  any  non-targets.  In  the  first 
case,  the  capabilities  of  the  automated  system  on  the  ADTS  were  identical  to 
those  of  the  ATS.  The  primary  benefit  would  be  the  requirement  for  fewer 
actions  to  handle  targets  that  the  AT  failed  to  track.  Given  the  use  of  a  relatively 
reliable  AT  in  this  study,  it  was  not  anticipated  that  this  would  result  in  much 
reduction  in  time  to  handle  the  targets.  In  the  second  case,  the  user  did  not  have 
to  worry  about  adding  targets  that  were  easily  detected.  Since  manual  detection  of 
signals  associated  with  strong  targets  was  relatively  easy,  it  was  not  expected  that 
performance  or  time  to  handle  the  targets  would  differ  substantially  from  that 
found  under  the  ATS  . 


*  Alternately,  the  user  could  remove  the  target  symbol,  add  the  signal  representing  the  target  at  its 
current  location  and  assign  it  to  the  AT.  However,  this  would  result  in  the  disappearance  of  the 
target  track. 
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METHOD 


Participants 

Six  people  (three  males  and  three  females)  participated  in  the  study.  Three 
of  the  participants  were  military  personnel,  while  the  remaining  were  civilian 
personnel  and  university  students.  They  ranged  in  age  from  21  to  43  years  and  all 
reported  normal  (20/20)  or  corrected-to-normal  vision. 


Apparatus 

The  experiment  was  carried  out  on  a  Power  Macintosh  with  a  17"  screen 
using  the  Automated  Tracking  Simulation  (ATS)  and  the  Automated  Detection 
and  Tracking  Simulation  (ADTS)  experimental  control  software^.  Both  the  ATS 
and  ADTS  are  simulations  of  a  target  tracking  task  for  studying  human  use  of  an 
automated  system.  Examples  of  the  ATS  and  ADTS  screens  are  shown  in  Figures 
1  and  2.  User  tasks  are  performed  through  a  series  of  tracking  display  (shown  on 
the  left  of  the  figures),  signal  table  (shown  on  the  right),  and  function  button  (far 
right)  selections.  All  selections  are  carried  out  via  a  mouse.  Invalid  selections  are 
distinguished  by  a  grey  shade,  beep,  or  no  action  following  a  selection. 

For  a  detailed  description  of  the  Automated  Tracking  System  (ATS)  refer 
to  McFadden,  Giesbrecht  and  Gula  (1997).  The  Automated  Detection  and 
Tracking  Simulation  (ADTS)  is  a  modified  version  of  the  ATS.  With  the  ATS, 
the  participant  has  the  option  of  giving  control  of  specific  targets  to  the  AT 
(Assign)  and  taking  away  control  (Deassign).  Signals  associated  with  targets 
under  AT  control  appear  in  the  AT  signal  table  which  is  accessed  by  clicking  on 
the  small  window  labelled  manual.  With  the  ADTS,  all  signals  can  be  handled 
all  the  time  by  either  the  human  or  the  ADT.  Thus  the  assign  and  deassign 
functions  are  no  longer  required  and  there  is  only  a  single  signal  table.  Signals 
that  are  associated  with  targets  are  highlighted.  Targets  that  are  associated  with 
signals  are  solid  black  and  targets  that  are  imassociated  have  a  white  X  in  them. 


^  Both  the  ATS  and  the  ADTS  were  developed  by  APG,  Toronto,  Ontario  under  contracts  from  the 
Defence  and  Civil  Institute  of  Environmental  Medicine. 
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Either  the  participant  or  the  ADT  can  add  and  associate  signals.  Only  the 
participant  can  deassociate  and  remove  targets. 


ik  Configuration  Session 


Session  Time 


180  manual  O  associated 

0  unassociated 

auto-tracker  #  associated 

X  unassociated 


Figure  1:  Schematic  of  the  display  for  the  Automated  Tracking  Simulation  (ATS) 
system.  The  "X'  for  the  tmassociated  automated  tracker  marker  appears 
as  a  white  X  in  a  black  circle  in  the  actual  display.  The  dashed  line  in 
sector  1  of  the  tracking  display  shows  the  path  traced  out  by  that  target. 
The  solid  line  at  the  top  of  the  same  target  shows  the  projected  direction 
for  that  target.  The  numbers  1-4  are  sector  labels  and  the  numbers  5  and 
10  distance  markers. 


Task 

The  participant's  task  is  to  track  the  location  of  various  vessels  (objects), 
given  a  set  of  signals.  An  automated  tracker  (AT)  can  be  activated  to  assist  in  this 
task.  The  ADTS  has  an  automated  detection  capability  as  well.  The  AT  and  ADT 
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(automated  detector  and  tracker),  when  activated,  act  as  filters  or  preprocessors 
for  the  manual  task.  The  position  of  the  targets  in  the  tracking  display  must  be 
updated  every  time  new  information  appears  in  the  signal  table.  With  the  ATS, 
this  task  can  be  carried  out  manually  or  the  human  can  assign  some  or  all  of  the 
targets  to  the  AT.  With  the  ADTS,  the  ADT  automatically  tries  to  update  the 
position  of  all  targets  previously  added  to  the  tracking  display  and  to  add  any 
new  signals  with  a  signal  strength  above  a  predefined  threshold.  If  the 
automated  system  fails  to  add  or  update  targets,  it  is  the  responsibility  of  the 
human  to  do  this  manually  before  new  information  arrives.  With  the  ATS, 
targets  assigned  to  the  AT  must  be  deassigned  before  they  can  be  updated 
manually.  With  the  ADTS,  this  step  is  not  necessary. 


^  Configuration  Session _ 
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Figure  2:  Schematic  of  the  display  for  the  Automated  Detection  and  Tracking 
System.  (See  Figure  1  for  further  explanation.) 


Target  sets 


Each  target  set  or  scenario  was  composed  of  several  different  targets.  Each 
target  followed  either  a  straight  line,  zigzag,  circular  or  experimenter-defined 
path.  The  scenarios  were  designed  so  that  the  number  of  targets  with  a  signal 
strength  below  20  was  usually  less  than  10%  on  average.  The  number  of  non¬ 
target  signals  added  on  each  update  ranged  from  2  to  8  and  their  signal  strength 
was  always  less  than  20. 

Conditions 

The  study  was  a  within-participant  design  with  two  independent  variables 
-  level  of  automation  and  number  of  targets.  Three  different  levels  were  used, 
automated  tracking  only,  automated  detection  and  tracking  with  the  detection 
threshold  set  at  100  so  that  no  targets  were  detected  (ADTS-100),  and  automated 
detection  and  tracking  with  the  threshold  set  at  20  so  that  no  non-targets  or  low 
strength  targets  were  detected  (ADTS-20).  The  second  variable  was  number  of 
targets,  either  15  or  20.  All  participants  completed  four  test  scenarios  under  each 
combination  of  conditions. 

The  automation  conditions  were  always  carried  out  in  order  of  increasing 
automation.  This  decision  was  made  to  offset  the  time  cost  associated  with 
running  these  types  of  studies  by  using  the  participants  and  their  data  from  a 
study  with  the  ATS.  Thus,  all  of  the  participants  had  initially  completed  an 
experiment  using  the  ATS.  Some  of  the  data  from  that  experiment  were  used  for 
the  ATS  condition.  The  remaining  two  conditions  could  have  been  randomized, 
but  for  consistency,  participants  carried  out  the  ADTS-100  condition  followed  by 
the  ADTS-  20.  The  ordering  of  the  number  of  targets  and  scenarios  was 
randomized  across  participants  within  each  level  of  automation  and  across  each 
level  of  automation  with  the  constraint  that  a  scenario  was  never  rim  twice  in 
the  same  session. 
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Procedure 


Participants  read  the  experimental  protocol  describing  the  background 
research,  the  task,  and  the  potential  risks  involved  in  participation.  Any 
questions  that  participants  had  about  the  study  were  answered  by  the 
experimenter  before  an  informed  consent  form  was  signed. 

During  training  and  test  sessions,  participants  were  seated  in  an  adjustable 
chair  at  the  normal  working  distance  from  the  computer  monitor 
(approximately  50  cm).  The  task  was  carried  out  in  normal  ambient 
illumination.  Each  session  lasted  from  1  to  1.5  hours  and  all  participants  were 
given  the  opportunity  to  take  breaks  between  runs  to  reduce  visual,  mental,  and 
postural  fatigue.  A  list  of  actions  that  could  be  carried  out  with  the  ATS  or  the 
ADTS  was  located  in  a  copy  holder  next  to  the  monitor  and  was  available  to 
participants  at  all  times  during  both  the  training  and  test  sessions.  As  well, 
participants  were  told  that  they  could  make  use  of  a  pencil  and  paper  situated  on 
the  computer  table  to  make  notes. 

Initially,  participants  carried  out  4  days  of  training  followed  by  four  test 
sessions  using  the  ATS.  On  the  first  day  of  training,  they  completed  five 
scenarios  in  which  they  had  to  monitor  four  targets  at  a  time  for  10  minutes. 
During  each  of  the  remaining  training  sessions  they  carried  out  four  scenarios  of 
16  minutes  duration.  On  days  two  and  three,  they  monitored  6  targets  at  a  time 
and  on  day  four,  they  monitored  eight.  During  the  test  sessions,  they  completed 
five  different  scenarios  with  10, 15,  and  20  targets. 

Prior  to  testing  on  the  ADTS,  the  participants  completed  an  additional 
training  session  composed  of  four  runs  to  allow  them  to  become  familiar  with 
the  changes  in  the  task  and  the  ADTS  interface.  The  participants  tracked  8  targets, 
using  the  scenarios  from  the  final  training  session  of  the  ATS  experiment.  A 
feedback  window  was  present  during  the  training  sessions  so  that  participants 
could  monitor  their  level  of  performance. 

The  training  session  on  the  ADTS  was  followed  by  four  test  sessions.  Test 
sessions  consisted  of  four  of  the  scenarios  used  m  the  ATS  sessions  with  15  and 
20  targets.  On  the  first  two  test  sessions,  participants  carried  out  these  eight 
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scenarios  under  the  ADTS-100  condition  and  on  the  third  and  fourth  sessions 
they  carried  out  the  same  scenarios  under  the  ADTS-20  condition. 


All  test  scenarios  or  runs  were  16  minutes  in  length,  with  updates  every  40 
seconds.  Five  target  signals  were  added  in  each  update  until  the  predefined 
maximum  was  reached  (15  or  20  targets).  A  solid  line  traced  out  the  last  five 
positions  (if  available)  of  the  target  marker  on  the  tracking  display  and  a  short 
line  indicated  the  projected  direction  and  speed  of  the  target  marker  based  on  the 
last  two  updates.  When  a  target  moved  out  of  the  range  of  the  tracking  display,  a 
new  target  was  added.  Throughout  all  the  test  sessions,  the  area  searched  by  the 
AT  was  set  to  60  and  the  probability  of  accepting  a  match  had  to  be  greater  than 
0.1  for  signals  with  strength  greater  than  10  and  0.6  for  lower  strength  signals. 
(The  number  60  is  strictly  a  function  of  the  experimental  control  program.) 
Preliminary  testing  showed  that  the  AT  would  track  80  to  90%  of  the  targets  in 
the  test  scenarios  when  those  AT  parameters  were  used. 

At  the  start  of  each  test  session,  participants  were  told  how  many  targets 
they  would  be  required  to  detect  and/or  track  (15  or  20)  to  the  best  of  their  ability. 
During  the  first  test  session,  they  were  also  informed  that  the  feedback  window 
was  no  longer  present  on  the  tracking  display.  Participants  were  encouraged  to 
take  short  breaks  between  rxms. 


RESULTS 

A  number  of  performance  measures  were  calculated  including  hit  rate, 
error  rates  for  the  different  types  of  errors  (misses,  lost  targets,  and 
misassociations),  false  alarm  rate,  the  percentage  of  signals  handled  manually 
and  by  the  ADT  and  the  amount  of  time  the  participants  spent  during  each 
update  handling  signals.  Performance  was  averaged  across  the  last  20  updates  for 
each  of  the  four  scenarios  under  the  six  different  experimental  conditions.  The 
first  four  updates  were  discarded  because  the  number  of  targets  in  the  20  target 
condition  did  not  reach  20  until  the  fourth  update. 

Data  from  the  ATS  rims  with  15  and  25  targets  that  used  the  four  scenarios 
run  in  the  ADTS  conditions  were  combined  with  the  ADTS  data  and  a  3  (ATS, 
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ADTS-100,  ADT-20)  x  2  (15  or  20  targets)  x  4  (scenarios)  within-participant 
analysis  of  variance  was  performed. 


Table  1  shows  average  performance  levels  for  the  various  measures.  In 
most  cases  performance  did  not  change  significantly  as  level  of  automation 
increased.  The  only  significant  changes  were  in  miss  and  false  alarm  rate.  Misses 
declined  significantly  F(2,10)  =  4.5,  p  >  0.05  with  a  posthoc  t-test  using  the  Tukey 
Studentized  Range  Test  indicating  that  the  miss  rate  for  ADTS-20  was 
significantly  lower  than  the  miss  rate  using  the  ATS.  Interestingly,  the  reduction 
in  percentage  of  misses  did  not  result  in  a  significant  improvement  in  hit  rate. 
This  probably  occurred  because  the  primary  source  of  error  in  all  conditions  was 
misassociations  and  that  error  rate  did  not  decrease  with  increasing  automation. 


Table  1:  Mean  time  to  handle  targets  and  percent  hits,  misassociations,  misses, 
losses,  and  false  alarms  as  a  function  of  task  and  number  of  targets. 


Automation 

level 

#of 

targets 

Time 

(sec) 

Hits 

Misasso¬ 

ciations 

Misses 

Losses 

False- 

alarms 

ATS 

15 

17.4 

79.1 

14.6 

5.2 

1.3 

06.4 

20 

22.2 

75.8 

14.0 

8.9 

1.5 

09.1 

ADT-100 

15 

13.3 

79.9 

15.5 

3.6 

0.9 

10.2 

20 

18.9 

78.1 

15.1 

6.0 

0.8 

15.6 

ADT-20 

15 

12.1 

81.8 

14.6 

2.9 

0.6 

10.7 

20 

18.1 

79.6 

15.8 

4.1 

0.4 

18.3 

The  other  measure  of  performance  that  changed  significantly  across  the 
three  conditions  was  false  alarm  rate,  F(2,10)  =7.2,  p  >  0.05.  In  this  instance,  there 
were  a  larger  number  of  false  alarms  in  the  ADTS  conditions  than  in  the  ATS.  A 
posthoc  t-test  showed  a  significant  difference  between  the  ATS  and  ADTS-20. 
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To  assess  the  source  of  the  false  alarms,  the  percentage  of  noise  signals 
handled  manually  and  by  the  AT  were  calculated  (Figure  3)  for  each  task.  Both 
manual  and  AT/ADT  generated  false  alarms  increased,  but  the  largest  increase 
was  in  the  number  of  manual  false  alarms. 

15.0 


10.0 

Percent 
false  alarms 

5.0 


0.0 

ATS  ADT-100  ADT-20 

Task 

Figure  3:  Comparison  of  percentage  of  false  alarms  generated  by  participants  and 
by  AT  imder  the  three  levels  of  automation. 


The  three  levels  of  automation  were  run  at  two  different  levels  of  task 
difficulty  which  was  defined  as  number  of  targets  that  had  to  be  monitored.  Our 
interest  was  in  whether  the  changes  in  the  task  had  a  differential  effect  as  a 
fvmction  of  task  difficulty.  As  shown  in  Table  1,  performance  declined  and 
response  time  increased  as  the  number  of  targets  that  had  to  be  monitored 
increased.  The  changes  in  response  time  and  hit  rate  were  both  significant,  F(2,5) 
=  105.7,  p  <  0.01  and  F(2,5)  =  6.6,  p  <  0.05  respectively.  The  decrease  in  hit  rate  was 
primarily  due  to  a  significant  increase  in  number  of  misses,  F(2,5)  =  58.6,  p  <  0.01. 
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False  alarm  rate  also  increased  significantly^  F(2,5)  =  73.8,  p  <  0.01.  However,  there 
was  no  interaction  between  task  t5^e  and  task  difficulty. 


DISCUSSION 

The  primary  purpose  of  this  study  was  to  determine  if  the  changes  in  the 
task  that  occurred  in  adding  an  automated  detector  affected  the  performance  of 
the  participants  in  any  way.  With  the  detection  threshold  of  the  ADT  set  to  100,  it 
had  been  anticipated  that  there  might  be  a  small  decrease  in  the  time  taken  to 
update  the  position  of  the  targets  because  fewer  operations  were  required  to 
correct  AT  errors.  With  the  threshold  set  to  20  a  further  slight  improvement  in 
response  time  might  be  expected.  The  ADT  would  add  high-strength  targets  that 
it  had  failed  to  update.  However,  since  the  lost  rate  was  already  very  low,  this 
was  not  expected  to  have  much  of  an  impact. 

Average  time  taken  per  update  did  decrease  an  average  of  4  seconds  with 
the  ADT  as  compared  to  the  ATS,  but  the  improvement  was  not  significant.  The 
significant  changes  that  did  occur  were  a  decrease  in  misses  and  an  increase  in 
false  alarms,  in  particular  manual  false  alarms,  in  both  ADT  conditions.  This 
would  be  consistent  with  the  participant  spending  more  time  to  find  low 
strength  targets.  Since  the  change  occurred  in  both  ADT  conditions,  it  appeared 
to  be  associated  more  with  the  change  in  task  rather  than  the  presence  of  an 
automated  detection  function  for  high  strength  targets. 

An  alternative  explanation  is  that  the  change  in  strategy  was  due  to 
experience.  All  the  participants  had  completed  several  sessions  using  the  ATS 
under  similar  conditions  before  they  started  the  sessions  with  the  ADTS.  It  may 
be  that  it  takes  considerable  experience  with  the  simulation  before  participants 
become  sufficiently  experienced  at  the  task  that  they  start  trying  new  strategies  to 
improve  their  performance.  With  a  reliable  AT,  the  primary  way  of  improving 
performance  is  to  reduce  the  number  of  missed  and  misassociated  targets. 

Without  feedback,  missed  targets  are  more  noticeable  than  misassociated  targets. 

Given  the  order  that  the  different  automation  conditions  were  carried  out 
in,  this  hypothesis  cannot  be  ruled  out.  However,  a  subset  of  the  participants  in 
the  original  experiment  with  the  ATS  also  participated  in  a  second  experiment 
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that  was  carried  out  on  the  ATS.  The  false  alarm  rates  in  all  three  studies  are 
shown  in  Figure  4.  As  will  be  noted,  the  false  alarm  rate  of  participants  in  the 
second  study  using  the  ATS  did  not  increase.  However,  they  only  had  to  monitor 
10  targets  while  those  in  the  study  using  the  ADT  monitored  either  15  or  20 
targets.  Those  in  the  ten  target  condition  may  not  have  felt  the  need  to  try 
alternative  strategies  to  improve  their  performance.  The  scenarios  were  designed 
so  the  percentage  of  low  strength  targets  was  usually  less  than  10%,  This 
translates  into  about  1  target  per  update  in  the  10  target  condition  and  two  targets 
per  update  in  the  20  target  condition.  That  could  account  for  the  difference  in 
false  alarm  rate  between  the  10  target  and  20  target  conditions.  On  the  other 
hand,  the  noise  signal  rate  was  constant  across  all  target  conditions.  Thus,  the 
probability  of  selecting  a  target  signal  out  of  the  non-target  signals  by  chance  was 
potentially  higher  in  the  15  and  20  target  conditions. 


20 


15 

Percent 
false  alarms 

10 

5 

0 


Figure  4:  Comparison  of  false  alarm  rate  in  first  experiment  with  ATS,  second 
experiment  with  ATS  and  experiment  with  ADTS  as  a  function  of 
number  of  targets.  (See  text  for  details). 


Experiment 

ATS-1  ATS-2  ATS-1  ADT  ATS-1  ADT 
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Overall  the  results  suggest  that  the  change  in  task  does  not  lead  to 
substantive  changes  in  performance  under  the  test  conditions  used  in  this  study. 
Further  research  is  required  to  determine  the  influence  of  the  threshold  level  of 
the  automated  detector  and  task  variables  such  as  number  of  targets  on  the 
usefulness  of  the  ADT. 


CONCLUSION 

An  experiment  was  run  to  determine  if  the  task  changes  that  resulted 
from  the  inclusion  of  an  automated  detection  function  in  a  simulation  designed 
to  study  the  use  and  usefulness  of  an  automated  tracker  would  have  a  significant 
impact  on  performance.  It  was  found  that  hit  rate  and  the  mean  time  taken  to 
handle  the  signals  on  each  update  did  not  change  significantly  as  a  result  of  the 
change  in  task.  Further  studies  are  required  to  determine  the  conditions  under 
which  the  addition  of  an  automated  detector  will  and  will  not  have  a  beneficial 
effect  on  these  types  of  performance  measures. 

Percentage  of  misses  declined  and  manually  induced  false  alarm  rate 
increased  significantly  suggesting  a  change  in  the  user's  strategy  when  carrying 
out  the  ADTS  task.  However,  an  alternative  hypothesis,  that  the  change  of 
strategy  was  due  to  the  increased  experience  level  of  the  participants  in  the  ADTS 
task,  could  not  be  ruled  out.  If  the  change  in  strategy  was  due  to  the  change  in 
task,  then  the  availability  of  an  automated  detection  function  might  lead  to  better 
performance  during  scenarios  with  a  large  number  of  weak  targets. 
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